$\mathcal F[X, F(X)]$ is just the fuction $\mathcal F \colon W \subseteq \mathbb R^2 \to \mathbb R$ applied to the argument $(X, F(X)) \in \mathbb R^2$. I think brackets were choosen by the author over paranthese to enhance readiability, that is
$$ \mathcal F[X, F(X)] = \mathcal F\bigl(X, F(X)\bigr) $$
$\mathcal F[X, F(X)]$ is just the fuction $\mathcal F \colon W \subseteq \mathbb R^2 \to \mathbb R$ applied to the argument $(X, F(X)) \in \mathbb R^2$. I think brackets were choosen by the author over paranthese to enhance readiability, that is $$ \mathcal F[X, F(X)] = \mathcal F\bigl(X, F(X)\bigr) $$