I wondered how the curl$$\text{rot}\mathbf{F}=\left( \begin{array}{ccc}\partial_y F_3-\partial_z F_2 \\ \partial_z F_1-\partial_x F_3 \\ \partial_x F_2-\partial_y F_1 \end{array} \right)$$of a vector field $\mathbf{F}=(F_1,F_2,F_3)$ changes when the basis $\mathbb{R}^3$ is changed. I would have thought that it is invariant, because of the intuitive and physical interpretation of the curl and because, if I correctly understand other quantities typical of vector fields, like divergence, are (if I am not wrong $\text{div}\mathbf{F}$ is the trace of the Jacobian matrix $J_{\mathbf{F}}$ of $\mathbf{F}$, and the trace of $E J_{\mathbf{F}} E^{\text{T}}$ - see below - is the same of $J_{\mathbf{F}}$), but I have got serious problem to prove it.
Trial:
If I am not wrong, if $E\in\text{O}(3)$ is the basis change matrix, and if we define the function $\mathbf{G}$ as $\mathbf{y}\mapsto E\mathbf{F}(^t E \mathbf{y})$, invariancy is equivalent to $$E\text{rot}\mathbf{F}=\text{rot}\mathbf{G}$$please correct me if I am wrong. The Jacobian matrix $J_{\mathbf{G}}(\mathbf{y})$ of $\mathbf{G}$ in $\mathbf{y}$ should be:$$J_{\mathbf{G}}(\mathbf{y})=E J_{\mathbf{F}}(\mathbf{x}) E^{\text{T}}$$where $E^{\text{T}}$ is the transpose, and inverse, matrix of $E$.
I have used such an identity to find the expression of $\text{rot}\mathbf{G}(\mathbf{y})$ in terms of the components of $J_{\mathbf{F}}(\mathbf{x})$ and $E$, but my calculations do not give me the expected result: for example, in the first component of the expression of $\text{rot}\mathbf{G}$ calculated in such a way , the coefficient of $\partial_x F_2$ is $(e_{21}e_{32}-e_{22}e_{31})$, while, in the first component of $E\text{rot}\mathbf{F}$, the coefficient of $\partial_x F_2$ is $e_{13}$...
Is the curl invariant under a change of orthogonal basis and, if it is, how can it be correctly proved? Thank you so much for any answer!
Update
In the first version of my answer (reprinted below) I tried to give a "geometrical" or "physical" explanation of the curl vector, but in fact did not address your question, which is the following: How do the coordinates of the curl vector ${\bf c}$ transform when we replace the given orthonormal coordinates $(x_i)$ by an equally oriented orthonormal coordinates $(\bar x_l)\,$?
Denote the transform matrix from the old to the new coordinates by $T=[t_{il}]\in SO(3)$. The columns of $T$ then are the old coordinates of the new basis vectors $\bar{\bf e}_l$, and the rows of $T$ are the new coordinates of the old basis vectors ${\bf e}_i$. In your setup the coordinates of the points ${\bf x}$ and those of the force field ${\bf F}$ transform in the same way according to formulas like $$x_i=\sum_l t_{il}\bar x_l,\quad \bar F_l=\sum_i t_{il} F_i\ .$$ Now in any orthonormal system the coordinates of the curl vector are given by $$c_i={\partial F_{i-1}\over\partial x_{i+1}}-{\partial F_{i+1}\over\partial x_{i-1}}\ .$$ In order to determine the new coordinates $\bar c_l$ of ${\bf c}$ we have to compute partial derivatives of the $\bar F_k$ using the chain rule: $${\partial \bar F_k\over\partial \bar x_l}=\sum_i t_{ik}\left(\sum_j{\partial F_i\over\partial x_j}\>{\partial x_j\over\partial\bar x_l}\right)=\sum_{i, \>j}{\partial F_i\over\partial x_j}\>t_{ik}t_{jl}\ .$$ We therefore obtain $$\bar c_l={\partial \bar F_{l-1}\over\partial\bar x_{l+1}}-{\partial \bar F_{l+1}\over\partial \bar x_{l-1}}=\sum_{i, \>j}{\partial F_i\over\partial x_j}\>(t_{i,l-1}t_{j,l+1}-t_{i,l+1}t_{j,l-1})\ .\tag{1}$$ Now $$t_{i,l-1}t_{j,l+1}-t_{i,l+1}t_{j,l-1}=({\bf e}_j\times {\bf e}_i)_l\ ,$$ which is $=0$ when $i=j$. This allows to write $(1)$ in the form $$\bar c_l=\sum_i {\partial F_{i-1}\over\partial x_{i+1}}({\bf e}_{i+1}\times {\bf e}_{i-1})_l+ \sum_i {\partial F_{i+1}\over\partial x_{i-1}}({\bf e}_{i-1}\times {\bf e}_{i+1})_l\ ,$$ or $$\bar c_l=\sum_i \left({\partial F_{i-1}\over\partial x_{i+1}}- {\partial F_{i+1}\over\partial x_{i-1}}\right)({\bf e}_i)_l=\sum_i c_i\>t_{il}\ .$$ This shows that the coordinates of the curl vector ${\bf c}$ transform in the same way as the coordinates of the vectors ${\bf x}$ and ${\bf F}$.
First version:
Given a force field ${\bf F}$ on a domain $\Omega\subset{\mathbb R}^3$ its curl measures the "local nonconservativity" of ${\bf F}$. Consider a point ${\bf p}\in\Omega$ and a small parallelogram $P$ of diameter $|P|$ with center ${\bf p}$, and spanned by two vectors ${\bf X}$ and ${\bf Y}$. Calculating the integral of ${\bf F}$ around $\partial P$ (which would be $=0$ when ${\bf F}$ is conservative) gives the following: $$\int_{\partial P}{\bf F}\cdot d{\bf x}= d{\bf F}({\bf p}).{\bf X}\cdot {\bf Y}- d{\bf F}({\bf p}).{\bf Y}\cdot {\bf X}+o\bigl(|P|^2\bigr)\ .$$ Here the main part of the right hand side is a skew bilinear function $R$ of the vectors ${\bf X}$ and ${\bf Y}$ spanning $P$: $$\int_{\partial P}{\bf F}\cdot d{\bf x}=R({\bf X},{\bf Y})+o\bigl(|P|^2\bigr)\ .$$ This setup makes sense in euclidean spaces of any dimension. Now in dimension $3$, such a skew bilinear function $R$ can be represented by a vector as follows: There is a well defined vector ${\bf c}$, called the curl of ${\bf F}$ at ${\bf p}$, such that $$\int_{\partial P}{\bf F}\cdot d{\bf x}={\bf c}\cdot({\bf X}\times{\bf Y})+o\bigl(|P|^2\bigr)\ .$$ My answer to your question would therefore be the following: As long as you stick to (linear) orthonormal coordinates of positive orientation the usual formula for ${\rm curl\,}{\bf F}$ is valid; but as soon as you introduce more general coordinates you would have to explain more carefully what you mean by the curl in the context.