I have been trying to figure out a way to calculate distance from camera (perspective projection) to a 3D point.
Currently I have depth buffer containing orthographic distances (perpendicular) from camera's plane (red line on the picture) to the given point. But I'm having hard time coming up with a working solution calculating distance from the camera to that point.

So from that picture I'd like to calculate the length of the purple line. We know the rotation of the plane (red line on the picture), direction vector of purple line and length of the blue line.
In the following example I refer to purple line's direction vector as 'dv'.
In the situation when camera is pointing towards -y -axis (see picture) the length of the purple line can be calculated: blue/-dv.y - ie. Dividing the length of the blue line by the length of direction vector's y component.
I have read about quaternions and would like to use quaternions in a working solution (not neccessary) but I'm not certainly sure how to achieve that. Any help is greatly appreciated.
There’s no need to bring quaternions into this. You’ve almost worked out the solution on your own already.
I’ll assume the standard camera coordinate system with the camera at the origin and pointed in the direction of the negative $z$-axis. I presume that, besides the depth, you have the focal distance $f$ and the coordinates $A'(x',y',f)$ of the projection of $A(x,y,z)$ onto the focal plane. These two points are on a common ray from the camera/eye, so if we let $B=(0,0,z)$ and $B'=(0,0,f)$, the triangles $\triangle{OAB}$ and $\triangle{OA'B'}$ are similar.
We have $A=\frac zfA'$ by the construction of the projection. This means that $|A|=|\frac zfA'|=|\frac zf|\sqrt{x'^2+y'^2+f^2}$, i.e., to recover the distance of $A$ from the camera, multiply the distance of its projection from the camera by the ratio of the orthographic distance to the focal distance.
Observe, too, that $|A|={z\over\cos\theta}$, where $\theta$ is the angle that the ray makes with the camera’s axis. Recall that for vectors $u$ and $v$, $u\cdot v=|u||v|\cos\theta$, which is just $\cos\theta$ when they are both unit vectors. So, if we have a unit vector $u$ that represents the camera’s facing and a unit vector $v$ that represents the direction of the ray, we can also compute the distance to $A$ as $|A|={z\over u\cdot v}$. This formulation can be handy if you’re not working in the standard camera coordinate system.