AB is a rod which is held such that $A=(1,-2,3)$ and $B=(2,3,-4)$ . A source of light is at the origin. Find the length of the shadow of the rod on a plane screen whose equation is $x+y+2z=1$
I figured out that origin and point B are on one side of given plane and point A is on other side. I found projection points of A and B and calculated distance between them but that is not matching with answer. Please help.

If what you say is correct, that the points A and B are on opposite sides of the plane, then the you answer will be wrong indeed, what you are looking for is not the projection of a vector representing the rod on the plane, but the shadow of the rod. Shadows are used to explain projections, but its a very particular analogy requiring specific circumstances. Draw it out for yourself and you'll see that actually the lenght you're looking for is the distance between the projection point of B on the plane, and the point where the the rod actually intersects the plane, as the part on the opposite side of the plane (opposite of the light source) won't cast a shadow on it