$\mathbf{a}\times \mathbf{b}$ follows the right hand rule? Why not left hand rule? Why is it $a b \sin (x)$ times the perpendicular vector? Why is $\sin (x)$ used with the vectors but $\cos(x)$ is a scalar product?
So why is cross product defined in the way that it is? I am mainly interested in the right hand rule defintion too as it is out of reach?
On
This may be a bit too deep but let $V$ be a finite dimensional vector space with basis $v_1,...,v_n$. We say $(v_1,...,v_n)$ is an oriented basis for $V$. We can define an equivalence class on orientations of $V$ by $[v_1,...,v_n] \sim [b_1,...,b_n] \iff [v_1,...,v_n] = A[b_1,...,b_n]$ (where $A$ is the transition matrix) and $\textbf{det}(A)>0$. Therefore, orientations are broken up into two classes, positive and negative. If we let $\textbf{e}^1,...,\textbf{e}^n$ be the standard basis for $\mathbb{R}^n$ then if we wish to check that $[b_1,...,b_n] \sim [\textbf{e}^1,...,\textbf{e}^n]$ then we simply have to look at;
$$\textbf{det}\left(\begin{bmatrix} b_1 & b_2 & \cdots & b_n \end{bmatrix}\right) >0$$
Since $A = [b_i]$ is the change of basis matrix.
$$\\$$
Edit (1): Above gives a generalization of determining orientation. In your case you would have $E=[\textbf{e}^1,\textbf{e}^2, \textbf{e}^3]$ which gives positive orientation and the ''right hand rule'' should be thought of as a geometric property that one can check without knowing all the information I presented above.
Edit (2): In regards to your question about why $\sin \theta, \cos \theta$ are used in certain situations is also because of geometry. The area of a parallelogram is base $\times$ height $= |\vec{a} \times \vec{b}| \sin \theta$ where $\vec{a}, \vec{b}$ span the parallelogram.
On
If the right hand rule seems too arbitrary to you, use a definition of the cross product that doesn't make use of it (explicitly). Here's one way to construct the cross product:
Recall that the (signed) volume of a parallelepiped in $\Bbb R^3$ with sides $a, b, c$ is given by
$$\textrm{Vol} = \det(a,b,c)$$
where $\det(a,b,c) := \begin{vmatrix}a_1 & b_1 & c_1 \\ a_2 & b_2 & c_2 \\ a_3 & b_3 & c_3\end{vmatrix}$.
Now let's fix $b$ and $c$ and allow $a$ to vary. Then what is the volume in terms of $a = (a_1, a_2, a_3)$? Let's see:
$$\begin{align}\textrm{Vol} = \begin{vmatrix}a_1 & b_1 & c_1 \\ a_2 & b_2 & c_2 \\ a_3 & b_3 & c_3\end{vmatrix} &= a_1\begin{vmatrix} b_2 & c_2 \\ b_3 & c_3\end{vmatrix} - a_2\begin{vmatrix} b_1 & c_1 \\ b_3 & c_3\end{vmatrix} + a_3\begin{vmatrix} b_1 & c_1 \\ b_2 & c_2\end{vmatrix} \\ &= a_1(b_2c_3-b_3c_2)+a_2(b_3c_1-b_1c_3)+a_3(b_1c_2-b_2c_1) \\ &= (a_1, a_2, a_3)\cdot (b_2c_3-b_3c_2,b_3c_1-b_1c_3,b_1c_2-b_2c_1)\end{align}$$
So apparently the volume of a parallelopiped will always be the vector $a$ dotted with this interesting vector $(b_2c_3-b_3c_2,b_3c_1-b_1c_3,b_1c_2-b_2c_1)$. We call that vector the cross product and denote it $b\times c$.
From the above construction we can define the cross product in either of two equivalent ways:
Implicit Definition
Let $b,c\in \Bbb R^3$. Then define the vector $d = b\times c$ by $$a\cdot d = \det(a,b,c),\qquad \forall a\in\Bbb R^3$$
Explicit Definition
Let $b=(b_1,b_2,b_3)$, $c=(c_1,c_2,c_3)$. Then define the vector $b\times c$ by $$b\times c = (b_2c_3-b_3c_2,b_3c_1-b_1c_3,b_1c_2-b_2c_1)$$
Now you're probably wondering where that arbitrary right-handedness went. Surely it must be hidden in there somewhere. It is. It's in the ordered basis I'm implicitly using to give the coordinates of each of my vectors. If you choose a right-handed coordinate system, then you'll get a right-handed cross product. If you choose a left-handed coordinate system, then you'll get a left-handed cross product. So this definition essentially shifts the choice of chirality onto the basis for the space. This is actually rather pleasing (at least to me).
The other properties of the cross product are readily verified from this definition. For instance, try checking that $b\times c$ is orthogonal to both $b$ and $c$. If you know the properties of determinants it should be immediately clear. Another property of the cross product, $\|b\times c\| = \|b\|\|c\|\sin(\theta)$, is easily determined by the geometry of our construction. Draw a picture and see if you can verify this one.
On
The cross product originally came from the quaternions, which extend the complex numbers with two other 'imaginary units' $j$ and $k$, that have noncommutative multiplication (i.e. you can have $uv \neq vu$), but satisfy the relations
$$ i^2 = j^2 = k^2 = ijk = -1 $$
AFAIK, this is the exact form that Hamilton originally conceived them. Presumably the choice that $ijk = -1$ is simply due to the convenience in writing this formula compactly, although it could have just as easily been an artifact of how he arrived at them.
Vector algebra comes from separating the quaternions into scalars (the real multiples of $1$) and vectors (the real linear combinations of $i$, $j$, and $k$). The cross product is literally just the vector component of the ordinary product of two vector quaternions. (the scalar component is the negative of the dot product)
The association of $i$, $j$, and $k$ to the unit vectors along the $x$, $y$, and $z$ axes is just lexicographic convenience; you're just associating them in alphabetic order.
On
the Gibbs Vector Algebra Cross Product Right Hand Rule is a Convention
Any description of the (nominally) Euclidean 3 dimensional world we perceive runs into Chirality, the distinction between Right and Left handedness, the difference in a Mirror Reflection. https://en.wikipedia.org/wiki/Chirality
A Mathematical System describing this world has to have a Convention for coordinate order, orientation.
Justifications for the Right Hand Rule for the Vector Cross Product are a consequence of the choice of Chirality in the the "Standard Basis" and are circular, not deep - you just have to choose.
In fact the other choice has been used: http://web.stanford.edu/class/me331b/documents/VectorBasisIndependent.pdf
The right-hand rule is a recently accepted universal convention, much like driving on the right-hand side of the road in North America. Until 1965, the Soviet Union used the left-hand rule, logically reasoning that the left-hand rule is more convenient because a right-handed person can simultaneously write while performing cross products.
On
The current Gibbs Vector Algebra System was invented by "Physicists" of the day - not "Pure Mathematicians" and its explicit purpose was describing Physical Phenomina in 3-D space - so it has to have a Convention for Handedness
http://worrydream.com/refs/Crowe-HistoryOfVectorAnalysis.pdf outlines the history, the book even more thoroughly, but I gave a try below
The Gibbs Vector Algebra was the winner of the "Quaternion Wars" in the late 19th century among Physicists as a useful system to describe Physical phenomena, in particular Electromagnetism, Heavisde essentially co-invented the same system.
At the beginning of the 19th century there were no Vector Algebras to choose from, Mathematics was not separate from Physics the way it may look today.
Math was being invented/made up/discovered by Natural Philosophers as they tried to describe the world in formal systems.
But what we now distinguish as Physics was advancing to describe more complicated phenomena like fluid flow in 3-D and writing everything in 3 sets of Cartesian component equations was limiting.
The patterns we now encode as Vector Dot and Cross Products can be seen in the Cartesian x, y, z equation systems of the early formulations of Fluid Mechanics and Maxwell's Equations.
Hamilton taking inspiration from the Gauss/Wessel/Argand Complex Number plane representation of 2-D space came up with Quaternions in 1843 and coined the term "Vector" for his "pure quaternion" 3 "imaginary" components.
Hamilton's influence from his many discovery's in Physics, Optics, Matrix Theory lead many to try to understand and use his Quaternion Algebra. Maxwell's Equations in particular became more compact compared to the original ~20 Cartesian component version.
But many found reasoning in the Hamilton Quaternion Algebra very unintuitive with his i, j, k basis unit elements squaring to -1.
This lead to Gibbs in the early 1880's "stealing" Hamilton's "Vector" name, pulling out the Dot and Cross Product from the Quaternion Multiplication rules and giving his system our current interpretation with x, y, z basis unit vectors each squaring to +1. "Closing" Gibbs new Vector Algebra required the Vector Cross Product we now know and use to give another Vector - but it really isn't mathematically correct as the Polar/Axial or Vector/PseudoVector distinction is needed in Gibbs Vector Algebra.
But both Hamilton and Gibbs had to choose an Orientation, basis metric and Order, impose a Convention on their systems to give agreement with the inherent Chirality of Euclidean 3-D space.
On
That the cross product has been defined as a vector at all is perhaps an accident of the fact that we perceive three spatial dimensions. It does not make sense in any other number of dimensions (e.g. 2 or 4, though some unnatural version can be constructed in 7). The exterior product (also called the wedge product) makes sense in any number of dimensions. It may be envisaged as an oriented surface with a direction of rotation, the area (or magnitude) of which is equal to the parallelogram defined by the vectors. These can be added and scaled, just like vectors. The existence of the cross product owes its existence to the isomorphism between the space of these products and the 3-dimensional vector space. It is simply the choice of considering these products (examples include torque, oriented area, angular momentum) to be normal vectors, though they are not (witness the fact that they are called pseudo-vectors). Their components do not scale with a change of basis (or units) as do vectors, but rather as their square. The need to choose of left- or right-handedness results entirely from the choice to map the product onto "normal" vectors, made possible by their isomorphism.
When the vectors are two-dimensional, the product inhabits a one dimensional space (torque in a two dimensions makes sense, so why insist on a third dimension to accommodate the normal vector?). In four dimensions, the equivalent of torque (the oriented product of a force and the length of a lever arm) inhabits a space of six dimensions, not four.
So, the short version is: it is a convention needed to deal with the unfortunate representation of the product of vectors as normal vectors. It would be completely (and more intuitively) avoided entirely if these products were thought of as oriented areas rather than as being classical vectors.
On
There's a whole algebra ("Geometric Algebra", or Clifford algebra) behind the concept of multiplying vectors. Look there to find out where the handedness comes from, why $ \cos \theta $ for $a \cdot b$ but $\sin \theta$ for $a \times b$, and many other gems. Here's a great reference.
Essentially, the cross product is a vector that describes a plane. A plane is described by two vectors (in any number of dimensions), but in $\mathbb{R}^3$ we have the handy situation that a plane can also be described by a vector. So, in 3-d only, vector $c = a \times b$ describes the plane described by $a$ and $b$. Since a plane has two sides, there's a two-fold arbitrariness ("orientation") that the choice of sign handles.
The $\sin$ and $\cos$ bit is slightly more complicated, but not much. Check out the reference.
On
As far as $\sin \theta$ and $\cos \theta$ are concerned,
Using the law of cosines,
\begin{align} \|\overrightarrow{v_2}\|^2 + \|\overrightarrow{v_1}\|^2 -2\|\overrightarrow{v_2}\| \, \|\overrightarrow{v_1}\| \cos \theta &= \|\overrightarrow{v_2} - \overrightarrow{v_1}\|^2 \\ 2x_1 x_2 + 2y_1 y_2 + 2z_1 z_2 &= 2\|\overrightarrow{v_2}\|\, \|\overrightarrow{v_1}\| \cos \theta \\ \cos \theta &= \dfrac{x_1 x_2 + y_1 y_2 + z_1 z_2} {\|\overrightarrow{v_2}\| \, \|\overrightarrow{v_1}\|} \\ \cos \theta &= \dfrac{\overrightarrow{v_2} \circ \overrightarrow{v_1}} {\|\overrightarrow{v_2}\| \, \|\overrightarrow{v_1}\|} \end{align}
And so, $\overrightarrow{v_1} \circ \overrightarrow{v_2} = \|\overrightarrow{v_2}\| \, \|\overrightarrow{v_1}\| \cos \theta$
Note that $\sin \theta$ is non negative for all $0 \le \theta \le \pi$.
So
\begin{align} \sin^2 \theta &= 1 - \cos^2 \theta \\ &= 1 - \dfrac{(x_1 x_2 + y_1 y_2 + z_1 z_2)^2} {\|\overrightarrow{v_2}\|^2 \, \|\overrightarrow{v_1}\|^2} \\ &= \dfrac{\|\overrightarrow{v_2}\|^2 \, \|\overrightarrow{v_1}\|^2 - (x_1 x_2 + y_1 y_2 + z_1 z_2)^2} {\|\overrightarrow{v_2}\|^2 \, \|\overrightarrow{v_1}\|^2} \\ &= \dfrac{x_2^2 y_1^2 - 2x_1 x_2 y_2 y_1 + x_1^2 y_2^2 + x_2^2 z_1^2 +x_1^2 z_2^2 - 2 x_1 x_2 z_1 z_2 +y_1^2 z_2^2-2 y_2 y_1 z_1 z_2+y_2^2 z_1^2} {\|\overrightarrow{v_2}\|^2 \, \|\overrightarrow{v_1}\|^2} \\ &= \dfrac{(-y_2 z_1+y_1 z_2)^2+(x_2 z_1-x_1 z_2)^2+(-x_2 y_1+x_1 y_2)^2} {\|\overrightarrow{v_2}\|^2 \, \|\overrightarrow{v_1}\|^2} \\ &= \dfrac {\|\overrightarrow{v_1} \times \overrightarrow{v_2} \|^2} {\|\overrightarrow{v_2}\|^2 \, \|\overrightarrow{v_1}\|^2} \end{align}
And we conclude, for purely formal reasons, $\|\overrightarrow{v_1} \times \overrightarrow{v_2} \| = \|\overrightarrow{v_2}\| \, \|\overrightarrow{v_1}\| \sin \theta$
On
The cross product doesn't actually follow any such rule. It is defined arithmetically as a determinant:
$$\mathbf{u\times v} = \begin{vmatrix} \mathbf{i}&\mathbf{j}&\mathbf{k}\\ u_1&u_2&u_3\\ v_1&v_2&v_3\\ \end{vmatrix}$$
There is no hand rule there. The right hand rule comes from the conventions about how we visualize the result in 3-D vector space.
Firstly, the resulting scalar value is assigned as the length of a vector which is orthogonal to $\mathbf{u}$ and $\mathbf{v}$. That does not give us a right hand rule yet.
If we identify the first vector component with the $x$ axis, the second with the $y$ axis and the third with the $z$ axis, such that the positive ray of the $x$ axis shoots left, $y$ shoots upward, and $z$ points out of the page toward us, then we end up with the right hand rule.
Why? Because the cross product of a unit vector lying in the positive $x$ axis direction, with a unit vector similarly lying in positive $y$ produces a unit vector lying in positive $z$.
Based on how we laid out the axes, this gives us the right hand rule: if we extend our thumb in the direction of $z$, then the fingers curl from $x$ to $y$.
Your question is really about why we lay out the axes this way, and that is merely a convention, like driving on the right side of the road (in the majority of the world's countries, not all, obviously).
The important thing about conventions is only a) to have them and b) have them be consistent. The choice between two equivalent conventions, like whether to choose one system or its exact mirror image, is unimportant; there is no external reason to go one way or the other.
On
As a complement to Bye_World's illuminating answer, which defines $b \times\ c$ as the vector whose dot product with $a$ equals $\det(a, b, c)$, $\|b\|\|c\|\sin(\theta)$ is the surface of the parallelogram formed by vectors $b$ and $c$. If $a$ is a unit vector orthogonal to the $b$-and-$c$ plane, then $\det(a, b, c)$, which is the (signed) volume of the parallelepiped formed by the 3 vectors, equals the surface of the $b$-and-$c$ parallelogram. In this case, i.e.$\|a\|=1$ and angle from $a$ to $b$ (and to $c$) equal to positive $\pi/2$, one can picture $b \times c$ as $a$ times the surface of the parallelogram.
On
I have answers for your question. If it was defined relative to the left-hand rule, you'd currently be asking why not the right-hand rule. It's how we deal with ambiguity: by establishing conventions. Depending on how you choose the definition of cross product, the result could be one of two different vectors. This led to the establishment of a convention. If there were no conventions with cross products, things would get very confusing because some people would be coming up with vectors in the opposite direction from other people, and their results would be different, which would lead to confusion. It's also likely that some theorems wouldn't work out because they're based on one definition and not the other, etc; one would keep having to convert between definitions all the time; and it would get very annoying.
The answer to your second questions? Upon establishing the definition of cross product, it just happened that the geometry worked out with sine and not cosine. It's just the behaviors of geometry in light of the constraints placed on it as a result having decided on a definition.
The same answer goes for the scalar product. Once the definition was made, that's how the geometry worked out.
However, it could be, though that the definitions were made so that they'd coincide with sin and cosine the way they did to unite a variety of approaches to the same set of problems, some people using trig, others using algebra to attack the same set of problems until some brilliant person united all the approaches with one definition. Such a thing often happens in mathematics, as anyone who's familiar with the concept of research might tell you.
Hope that helps.
Well, one can see the rule as is in order to agree with the standard orientation of $\mathbb{R}^3$ $(e_1 \times e_2=e_3)$. Or just convention.
There is a way to see how this can come up "naturally", though.
Given $a,b \in \mathbb{R}^3$, $a \times b$ is the vector which comes out from Riesz representation theorem (a fancy way to say that $V \simeq V^*$ using the isomorphism given by the inner product) applied to the linear functional
$$\det(\cdot, a,b).$$
Computing the vector yields both the fact that it has its norm being what it should be and its direction also being what it should be.
To get to the other direction, we would have to invert $a,b$, considering $a \times b$ to come from the vector corresponding to $\det(\cdot, b,a)$, an "unnatural" thing.