I am using quaternions to rotate my camera. I only rotating the camera around it's x and y axises. But for some reason, after some rotation, y axis also gets rotated(looks like around z axis).
I could avoid the problem with fixing y to (0,1,0), but I am curious where I am wrong.
On
An answer you would probably receive in a game dev forum:
It sounds like your using a Y up system and pitching around the world X axis. Typically, for a game camera where you only want yaw & pitch, usually the yaw is desired about the world's vertical axis but pitch is desired about the camera's local horizontal axis(local X axis). When the camera orientation is in the identity orientation, the camera's local horizontal axis happens to be aligned with the world horizontal axis (usually the X axis) but when yaw is applied the local horizontal axis rotates away from the world X axis. If you then apply pitch around the world X axis, it can cause a rotation with a Z axis component to it. But if you apply the pitch rotation first (before the yaw) while the local and world X axes are aligned, there will be no Z component to the rotation.
This pitch rotation causes the local Up axis to rotate away from the world Up axis too but you really don't care because you always yaw around the world up axis anyway.
So try applying pitch first, then yaw and see if it helps.
or... calculate the local horizontal axis after yawing and use that axis to pitch around instead of the world X axis.
How are you handling rotations? If you're composing rotations then you might be running into the fundamental 'trick' with rotations: their non-commutativity. In particular, if you have some rotation $R$ that's built (e.g.) as 'rotate around the $x$ axis $10^{\circ}$, then rotate around the $y$ axis $10^{\circ}$', then composing $R$ with itself (i.e., 'rotate around the $x$ axis $10^{\circ}$, then rotate around the $y$ axis $10^{\circ}$, then rotate around $x$ $10^{\circ}$ again, then rotate around $y$ $10^{\circ}$ again) is not the same as 'rotate around the $x$ axis $20^{\circ}$, then rotate around the $y$ axis $20^{\circ}$'! (This happens because 'rotate around $x$ by $10^{\circ}$, then rotate around $y$ by $10^{\circ}$' isn't the same as 'rotate around $y$ by $10^{\circ}$, then rotate around $x$ by $10^{\circ}$'.) If you want to keep your camera upright and you're not allowing arbitrary camera rotations, then it's a good idea to explicitly consider your camera in terms of the classic Yaw/Pitch/Roll Euler angles and manipulate those rather than by composing rotations.
Incidentally, for questions like this (practical questions about the implementation of rotation of objects, rather than abstract questions about the space of rotations) I'd recommend the gamedev StackExchange site; this is exactly the sort of thing they handle on a regular basis.