The equation has some notation that is difficult to find the meaning for. It is equation (3) in the paper 'Quaternion Averaging' by F. Landis Markley, et al. on page 3 under 'The Average Quaternion'. Here is the equation (forgive the poor Latex):
Using the definition of the Frobenius norm, the orthogonality of $A(\mathbf{q})$ and $A(\mathbf{q}_i)$, and some properties of the matrix trace (denoted by $\text{Tr}$) gives
$$ \begin{eqnarray} \lVert A(\mathbf{q}) - A(\mathbf{q}_i) \rVert ^2_F && = \text{Tr } \{[A(\mathbf{q}) - A(\mathbf{q}_i)]^T[A(\mathbf{q}) - A(\mathbf{q}_i)]\} \\ && = 6-2 \text{ Tr }[A(\mathbf{q})A^T(\mathbf{q}_i)] \end{eqnarray} $$
Specifically, with reference to $A(\mathbf{q})$, the brackets suggest this is a function, but $A$ would usually be the symbol for a matrix. $\mathbf{q}$ is likely a quaternion, so maybe a quaternion matrix?
Also, with reference to $A(\mathbf{q}_i)$, the $_i$ would be used in summation, but there is a lack of summation in the equation itself that would use it. Elsewhere in the paper it suggests $\mathbf{q}_i$ is a set of n attitude estimates, so a set of quaternions?
Curly braces would usually denote a set, but is it hard to tell whether ${[A(\mathbf{q}) - A(\mathbf{q}_i)]^T[A(\mathbf{q}) - A(\mathbf{q}_i)]}$ is a set.
Likewise, square brackets may denote any number of things, including equivalence class, or the floor, and so on.
Each $\mathbf q_i$ is an attitude estimate ("in quaternion form," so I'd say it is a quaternion).
$A(\mathbf{q})$ is a matrix. More specifically, the paper says on page 3, it is the attitude matrix of a quaternion $\mathbf{q}$.
The fact that the equation doesn't have a summation and the use of an arbitrary index $i$ also tell you that the expression is valid for any attitude estimate. The equation is plugged into a summation in equation (2).
The expression $[A(\mathbf{q}) - A(\mathbf{q}_i)]^T[A(\mathbf{q}) - A(\mathbf{q}_i)]$ is the product of two matrices; one is the transpose of the matrix $A(\mathbf{q}) - A(\mathbf{q}_i)$ and the other is the matrix $A(\mathbf{q}) - A(\mathbf{q}_i)$. Square brackets are used, I think, to avoid nested parentheses.
The trace is a function of a matrix; a reason for the use of the curly braces could be that there are parenthesis and square brackets in the argument.
I found another paper that explains the notation (page 2, especially): http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20030093641.pdf